Revisiting Natural Gradient for Deep Networks

We evaluate natural gradient, an algorithm originally proposed in Amari (1997), for learning deep models. The contributions of this paper are as follows. We show the connection between natural gradient and three other recently proposed methods: Hessian-Free (Martens, 2010), Krylov Subspace Descent (Vinyals and Povey, 2012) and TONGA (Le Roux et al., 2008). We empirically evaluate the robustness of natural gradient to the ordering of the training set compared to stochastic gradient descent and show how unlabeled data can be used to improve generalization error. Another contribution is to extend natural gradient to incorporate second error information alongside the manifold information. Lastly we benchmark this new algorithm as well as natural gradient, where both are implemented using a truncated Newton approach for inverting the metric matrix instead of using a diagonal approximation of it.